---
title: Plugins
description: Meltano takes a modular approach to data engineering and EL(T), where your project and pipelines are composed of plugins.
layout: doc
redirect_from:
  - /guide/analysis
sidebar_position: 2
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

Meltano takes a modular approach to data engineering in general and EL(T) in particular,
where your [project](/concepts/project) and pipelines are composed of plugins of [different types](#types), most notably
[**extractors**](#extractors) ([Singer](https://singer.io) taps),
[**loaders**](#loaders) ([Singer](https://singer.io) targets),
[**utilities**](#utilities) ([dbt](https://www.getdbt.com) for transformation, [Airflow](https://airflow.apache.org/)/[Dagster](https://dagster.io/)/etc. for orchestration, and much more on [MeltanoHub](https://hub.meltano.com/utilities/)).

Meltano provides the glue to make these components work together smoothly and enables consistent [configuration](/guide/configuration) and [deployment](/guide/production).

To learn how to manage your project's plugins, refer to the [Plugin Management guide](/guide/plugin-management).

## Project Plugins

In order to use a given package as a plugin in a [project](/concepts/project),
assuming it meets the requirements of the [plugin type](#types) in question, Meltano needs to know:

1. where to find the package, typically a [pip package](https://pip.pypa.io/en/stable/) identified by its name on [PyPI](https://pypi.org/), public or private Git repository URL, or local directory path,
2. what [settings](/guide/configuration) it supports
3. what [capabilities](/reference/plugin-definition-syntax#capabilities) it supports, and finally
4. what its [configuration](/guide/configuration) should be when invoked.

Together, a package's location (1) and the metadata (2) describing it in terms Meltano can understand make up the **base plugin description**.
In your project, **plugins** extend this description with a specific configuration (3) and a unique name.

This means that [different configurations](/guide/configuration#multiple-plugin-configurations) of the same package (base plugin)
would be represented in your project as separate plugins with their own unique names,
that can be thought of as differently initialized instances of the same class.
For example: extractors `tap-postgres--billing` and `tap-postgres--events` derived from base extractor [`tap-postgres`](https://hub.meltano.com/extractors/postgres.html),
or `tap-google-analytics--client-foo` and `tap-google-analytics--client-bar` derived from base extractor [`tap-google-analytics`](https://hub.meltano.com/extractors/google-analytics.html).

Each plugin in a project can either:

- inherit its base plugin description from a [discoverable plugin](#discoverable-plugins) that's supported out of the box,
- define its base plugin description explicitly, making it a [custom plugin](#custom-plugins), or
- [inherit](#plugin-inheritance) both base plugin description and configuration from another plugin in the project.

To learn how to add a plugin to your project, refer to the [Plugin Management guide](/guide/plugin-management#adding-a-plugin-to-your-project).

## Discoverable plugins

[Base plugin descriptions](#project-plugins) for many popular [extractors](#extractors) (Singer taps), [loaders](#loaders) (Singer targets),
and other plugins have already been collected by users and [contributed](/contribute/plugins#discoverable-plugins) to [Meltano Hub](https://hub.meltano.com),
making them supported out of the box.

To find discoverable plugins refer to the lists of [Extractors](https://hub.meltano.com/extractors/), [Loaders](https://hub.meltano.com/loaders/), etc., on [Meltano Hub](https://hub.meltano.com/).

To learn how to add a discoverable plugin to your project using a [shadowing plugin definition](/concepts/project#shadowing-plugin-definitions) or [inheriting plugin definition](/concepts/project#inheriting-plugin-definitions), refer to the [Plugin Management guide](/guide/plugin-management#discoverable-plugins).

### Variants

In the case of various popular data sources and destinations, multiple alternative implementations of Singer taps ([extractors](#extractors)) and targets ([loaders](#loaders)) exist,
some of which are forks of an original (canonical) version that evolved in their own direction, while others were developed independently from the start.

These different implementations and their repositories typically use the same name (`tap-<source>` or `target-<destination>`)
and may on the surface appear interchangeable, but often vary significantly in terms of exact behavior, quality, and supported settings.

In its [index of discoverable plugins](#discoverable-plugins), Meltano considers these different implementations different _variants_ of the same plugin,
that share a plugin name and other source/destination-specific details (like a logo and description),
but have their own implementation-specific variant name and metadata (like capabilities and settings).

Every discoverable plugin has a default variant that is known to work well and recommended for new users,
which will be added to your project unless you explicitly select a different one.
Users who already have experience with a different variant (or have specific reasons to prefer it) can explicitly choose to add it to their project instead of the default,
so that they get the same behavior and can use the same settings as before.
If the variant in question is not discoverable yet, it can be added as a [custom plugin](#custom-plugins).

To learn how to add a non-default variant of a discoverable plugin to your project, refer to the [Plugin Management guide](/guide/plugin-management#variants).

## Custom plugins

If you'd like to use a package in your project whose [base plugin description](#project-plugins) isn't [discoverable](#discoverable-plugins) yet,
you'll need to collect and provide this metadata yourself.

To learn how to add a custom plugin to your project using a [custom plugin definition](/concepts/project#custom-plugin-definitions), refer to the [Plugin Management guide](/guide/plugin-management#custom-plugins).

:::caution

  <p>Once you've got the plugin working in your project, please consider <a href="/contribute/plugins#discoverable-plugins">contributing its description</a> to <a href="https://github.com/meltano/hub/issues/new">Meltano Hub</a> to make it discoverable and supported out of the box for new users!</p>
:::

## Plugin Inheritance

If you'd like to use the same package (base plugin) in your project multiple times with [different configurations](/guide/configuration#multiple-plugin-configurations),
you can add a new plugin that inherits from an existing one.

The new plugin will inherit its parent's [base plugin description](#project-plugins) and [configuration](/guide/configuration) as if they were defaults,
which can then be overridden as appropriate.

For performance reasons, inherited plugins with an identical `pip_url` to their parent share the parents underlying python virtualenv.
If you would prefer to create a separate virtualenv for an inherited plugin, modify it's `pip_url` to be different to its parent.

To learn how to add an inheriting plugin to your project using an [inheriting plugin definition](/concepts/project#inheriting-plugin-definitions), refer to the [Plugin Management guide](/guide/plugin-management#plugin-inheritance).

## Lock artifacts

When you add a plugin to your project using `meltano add`, the [discoverable plugin definition](#discoverable-plugins) of the plugin will be downloaded and added to your project under `plugins/<plugin_type>/<plugin_name>--<variant_name>.lock`. This will ensure that the plugin's definition will be stable and version-controlled.

Later invocations of the plugin will use this file to determine the settings, installation source, etc.

:::info

  <p>Note that <a href="#custom-plugins">custom</a> and <a href="#plugin-inheritance">inherited</a> plugins do not get a lock file.</p>
:::

## Types

Meltano supports the following types of plugins:

- [**Extractors**](#extractors) pull data out of arbitrary data sources.
- [**Mappers**](#mappers) perform stream map transforms on data between extractors and loaders.
- [**Loaders**](#loaders) load extracted data into arbitrary data destinations.
- [**Utilities**](#utilities) perform arbitrary tasks provided by [pip packages](https://pip.pypa.io/en/stable/) with executables. All plugins previously referred to as `transformers` and `orchestrators` are being transistioned to utilities.
- [**File bundles**](#file-bundles) bundle files you may want in your project.

These plugin types are still supported but are transitioning to being referred to as [**Utilities**](#utilities):

- [**Orchestrators**](#orchestrators) orchestrate a project's scheduled pipelines.
- [**Transformers**](#transformers) run transforms.

These plugin types are deprecated:

- [**Transforms**](#transforms) transform data that has been loaded into a database (data warehouse).

### Extractors

Extractors are [pip packages](https://pip.pypa.io/en/stable/) used by [`meltano run`](/reference/command-line-interface#run) or [`meltano invoke`](/reference/command-line-interface#invoke) as part of [data integration](/guide/integration).
They are responsible for pulling data out of arbitrary data sources: databases, SaaS APIs, or file formats.

Meltano supports [Singer taps](https://singer.io): executables that implement the [Singer specification](https://hub.meltano.com/singer/spec).

To learn which extractors are [discoverable](#discoverable-plugins) and supported out of the box, refer to the [Extractors page](https://hub.meltano.com/extractors/).

#### Extras

Extractors support the following [extras](/guide/configuration#plugin-extras):

- [`catalog`](#catalog-extra)
- [`load_schema`](#load-schema-extra)
- [`metadata`](#metadata-extra)
- [`schema`](#schema-extra)
- [`select`](#select-extra)
- [`select_filter`](#select-filter-extra)
- [`state`](#state-extra)
- [`use_cached_catalog`](#cache-catalog-extra)

#### `catalog` extra

- Setting: `_catalog`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__CATALOG`, e.g. `TAP_GITLAB__CATALOG`
- [`meltano el`](/reference/command-line-interface#el) CLI option: `--catalog`
- Default: None

An extractor's `catalog` [extra](/guide/configuration#plugin-extras) holds a path to a [catalog file](https://hub.meltano.com/singer/spec#catalog-files) (relative to the [project directory](/concepts/project)) to be provided to the extractor
when it is run in [sync mode](https://github.com/singer-io/getting-started/blob/master/docs/SYNC_MODE.md) using [`meltano el`](/reference/command-line-interface#el) or [`meltano invoke`](/reference/command-line-interface#invoke).

If a catalog path is not set, the catalog will be [generated on the fly](/guide/integration#extractor-catalog-generation)
by running the extractor in [discovery mode](https://hub.meltano.com/singer/spec#discovery-mode) and applying the [schema](#schema-extra), [selection](#select-extra), and [metadata](#metadata-extra) rules to the discovered file.

[Selection filter rules](#select-filter-extra) are always applied to manually provided catalogs as well as discovered ones.

While this extra can be managed using [`meltano config`](/reference/command-line-interface#config) or environment variables like any other setting,
a catalog file is typically provided using [`meltano el`](/reference/command-line-interface#el)'s `--catalog` option.

If the catalog does not seem to take effect, you may need to [validate the capabilities of the tap](/guide/integration#validate-tap-capabilities).

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-gitlab
  catalog: extract/tap-gitlab.catalog.json
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _catalog <path>

meltano el <extractor> <loader> --catalog <path>

# For example:
meltano config tap-gitlab set _catalog extract/tap-gitlab.catalog.json

meltano el tap-gitlab target-jsonl --catalog extract/tap-gitlab.catalog.json

```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__CATALOG=<path>

# For example:
export TAP_GITLAB__CATALOG=extract/tap-gitlab.catalog.json
```

  </TabItem>
</Tabs>

#### <a name="load-schema-extra"></a>`load_schema` extra

- Setting: `_load_schema`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__LOAD_SCHEMA`, e.g. `TAP_GITLAB__LOAD_SCHEMA`
- Default: `$MELTANO_EXTRACTOR_NAMESPACE`, which [will expand to](/guide/configuration#expansion-in-setting-values) the extractor's `namespace`, e.g. `tap_gitlab` for `tap-gitlab`

An extractor's `load_schema` [extra](/guide/configuration#plugin-extras)
holds the name of the database schema extracted data should be loaded into,
when this extractor is used in a pipeline with a [loader](#loaders) for a database that supports schemas, like [PostgreSQL](https://www.postgresql.org/docs/current/ddl-schemas.html) or [Snowflake](https://docs.snowflake.com/en/sql-reference/ddl-database.html).

The value of this extra [can be referenced](/guide/configuration#expansion-in-setting-values) from a loader's configuration using the `MELTANO_EXTRACT__LOAD_SCHEMA`
[pipeline environment variable](/guide/integration#pipeline-environment-variables).
It is used as the default value for the [`target-postgres`](https://hub.meltano.com/loaders/postgres.html) and [`target-snowflake`](https://hub.meltano.com/loaders/snowflake.html) `schema` settings.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-gitlab
  load_schema: gitlab_data
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _load_schema <schema>

# For example:
meltano config tap-gitlab set _load_schema gitlab_data

```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__LOAD_SCHEMA=<schema>

# For example:
export TAP_GITLAB__LOAD_SCHEMA=gitlab_data
```

  </TabItem>
</Tabs>

#### `metadata` extra

- Setting: `_metadata`, alias: `metadata`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__METADATA`, e.g. `TAP_GITLAB__METADATA`
- Default: `{}` (an empty object)

An extractor's `metadata` [extra](/guide/configuration#plugin-extras) holds an object describing
[Singer stream and property metadata](https://hub.meltano.com/singer/spec#metadata)
rules that are applied to the extractor's [discovered catalog file](https://hub.meltano.com/singer/spec#catalog-files)
when the extractor is run using [`meltano run`](/reference/command-line-interface#run), [`meltano invoke`](/reference/command-line-interface#invoke), or [`meltano el`](/reference/command-line-interface#el).
These rules are not applied when a catalog is [provided manually](#catalog-extra).

Stream (entity) metadata `<key>: <value>` pairs (e.g. `{"replication-method": "INCREMENTAL"}`) are nested under top-level entity identifiers that correspond to Singer stream `tap_stream_id` values.
These [nested properties](/reference/command-line-interface#nested-properties) can also be thought of and interacted with as settings named `_metadata.<entity>.<key>`.

Property metadata `<key>: <value>` pairs (e.g. `{"is-replication-key": true}`) are nested under top-level stream identifiers and second-level property identifiers that correspond to Singer stream property names.
These [nested properties](/reference/command-line-interface#nested-properties) can also be thought of and interacted with as settings named `_metadata.<stream>.<property>.<key>`.

[Unix shell-style wildcards](<https://en.wikipedia.org/wiki/Glob_(programming)#Syntax>) can be used in stream and property identifiers to match multiple streams and/or properties at once.

Stream and property names can be discovered using [`meltano select --list --all <plugin>`](/reference/command-line-interface#select).

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-postgres
  metadata:
    some_stream_id:
      replication-method: INCREMENTAL
      replication-key: created_at
      created_at:
        is-replication-key: true
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _metadata <stream> <key> <value>
meltano config <extractor> set _metadata <stream> <property> <key> <value>

# For example:
meltano config tap-postgres set _metadata some_stream_id replication-method INCREMENTAL
meltano config tap-postgres set _metadata some_stream_id replication-key created_at
meltano config tap-postgres set _metadata some_stream_id created_at is-replication-key true

```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__METADATA='{"<stream>": {"<key>": "<value>", "<property>": {"<key>": "<value>"}}}'

# Once metadata has been set in `meltano.yml`, environment variables can be used
# to override specific nested properties:
export <EXTRACTOR>__METADATA_<ENTITY>_<ATTRIBUTE>_<KEY>=<value>

# For example:
export TAP_POSTGRES__METADATA_SOME_STREAM_ID_REPLICATION_METHOD=INCREMENTAL
export TAP_POSTGRES__METADATA_SOME_STREAM_ID_REPLICATION_KEY=created_at
```

  </TabItem>
</Tabs>

Common metadata keys that can be set include:

- `key-properties`: A list of properties that together uniquely identify a record in the stream. For example, `["id"]`.
- `replication-key`: The name of a property in the source to use as a bookmark. For example, this will often be an `"updated_at"` field or an auto-incrementing primary key (requires `replication-method`).
- `replication-method`: The replication method to use for a stream. Either `FULL_TABLE` or `INCREMENTAL`. Some extractors also support `LOG_BASED`, so check the extractor's documentation for details.

#### `schema` extra

- Setting: `_schema`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__SCHEMA`, e.g. `TAP_GITLAB__SCHEMA`
- Default: `{}` (an empty object)

An extractor's `schema` [extra](/guide/configuration#plugin-extras) holds an object describing
[Singer stream schema](https://hub.meltano.com/singer/spec#schemas) override
rules that are applied to the extractor's [discovered catalog file](https://hub.meltano.com/singer/spec#catalog-files)
when the extractor is run using [`meltano el`](/reference/command-line-interface#el) or [`meltano invoke`](/reference/command-line-interface#invoke).
These rules are not applied when a catalog is [provided manually](#catalog-extra).

[JSON Schema](https://json-schema.org/) descriptions for specific properties (e.g. `{"type": ["string", "null"], "format": "date-time"}`) are nested under top-level stream identifiers that correspond to Singer stream `tap_stream_id` values, and second-level property identifiers that correspond to Singer stream property names.
These [nested properties](/reference/command-line-interface#nested-properties) can also be thought of and interacted with as settings named `_schema.<stream>.<property>` and `_schema.<stream>.<property>.<key>`.

[Unix shell-style wildcards](<https://en.wikipedia.org/wiki/Glob_(programming)#Syntax>) can be used in stream and property identifiers to match multiple streams and/or properties at once.

Stream and property names can be discovered using [`meltano select --list --all <plugin>`](/reference/command-line-interface#select).

If a schema is specified for a property that does not yet exist in the discovered stream's schema, the property (and its schema) will be added to the catalog.
This allows you to define a full schema for taps such as [`tap-dynamodb`](https://github.com/singer-io/tap-dynamodb) that do not themselves have the ability to discover the schema of their streams.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-postgres
  schema:
    some_stream_id:
      created_at:
        type: ["string", "null"]
        format: date-time
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _schema <entity> <attribute> <schema description>
meltano config <extractor> set _schema <entity> <attribute> <key> <value>

# For example:
meltano config tap-postgres set _metadata some_stream_id created_at type '["string", "null"]'
meltano config tap-postgres set _metadata some_stream_id created_at format date-time
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__SCHEMA='{"<entity>": {"<attribute>": {"<key>": "<value>"}}}'

# Once schema descriptions have been set in `meltano.yml`, environment variables can be used
# to override specific nested properties:
export <EXTRACTOR>__SCHEMA_<ENTITY>_<ATTRIBUTE>_<KEY>=<value>

# For example:
export TAP_POSTGRES__SCHEMA_SOME_STREAM_ID_CREATED_AT_FORMAT=date
```

  </TabItem>
</Tabs>

#### `select` extra

- Setting: `_select`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__SELECT`, e.g. `TAP_GITLAB__SELECT`
- Default: `["*.*"]`

An extractor's `select` [extra](/guide/configuration#plugin-extras) holds an array of [stream and property selection rules](/reference/command-line-interface#select)
that are applied to the extractor's [discovered catalog file](https://hub.meltano.com/singer/spec#catalog-files)
when the extractor is run using [`meltano run`](/reference/command-line-interface#run), [`meltano invoke`](/reference/command-line-interface#invoke), or [`meltano el`](/reference/command-line-interface#el).
These rules are not applied when a catalog is [provided manually](#catalog-extra).

The `select` extra defines what streams and properties within each stream should be included in the extraction.

A selection rule can be either:
- **Stream-only pattern** (e.g., `"users"`): Selects the stream and all its properties, equivalent to `"users.*"`
- **Stream.property pattern** (e.g., `"users.id"`, `"orders.total"`): Selects specific properties within a stream

Rules indicating that an stream or property should be excluded are prefixed with an exclamation mark (`!`). [Unix shell-style wildcards](<https://en.wikipedia.org/wiki/Glob_(programming)#Syntax>) can be used in stream and property identifiers to match multiple entities and/or properties at once.

Entity and property names can be discovered using [`meltano select --list --all <plugin>`](/reference/command-line-interface#select).

While this extra can be managed using [`meltano config`](/reference/command-line-interface#config) or environment variables like any other setting,
selection rules are typically specified using [`meltano select`](/reference/command-line-interface#select).

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-gitlab
  select:
  - project_members      # Stream-only: includes all properties (same as project_members.*)
  - commits.id           # Specific property: only the id field from commits
  - commits.author_name  # Specific property: only the author_name field from commits
  - issues.*             # Explicit wildcard: all properties from issues stream
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _select '["<entity>.<attribute>", ...]'

meltano select <extractor> <entity> <attribute>

# For example:
# Stream-only patterns (both lines are equivalent)
meltano config tap-gitlab set _select '["project_members", "commits"]'
meltano config tap-gitlab set _select '["project_members.*", "commits.*"]'

# Mix of stream-only and specific properties
meltano config tap-gitlab set _select '["project_members", "commits.id", "commits.author_name"]'

# Using meltano select command
meltano select tap-gitlab project_members "*"  # All properties
meltano select tap-gitlab commits id           # Specific property
meltano select tap-gitlab commits author_name  # Another specific property

```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__SELECT='["<entity>.<attribute>", ...]'

# For example:
# Stream-only patterns (equivalent to wildcards)
export TAP_GITLAB__SELECT='["project_members", "commits"]'
# Mix of patterns
export TAP_GITLAB__SELECT='["project_members", "commits.id", "commits.author_name"]'
```

  </TabItem>
</Tabs>


If the extractor uses a dot character within stream names, you can escape it with a backslash (`\`). For example, if the extractor has a stream named `animals.data`, you can select fields using the following configuration:


<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-smoke-test
  select:
  - "animals\\.data.id"     # Use double backslash to escape the dot in quoted strings
  - animals\.data.verified  # Use single backslash to escape the dot in unquoted strings
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config tap-smoke-test set _select '["animals\\.data.id","animals\\.data.verified"]'

meltano select tap-smoke-test 'animals\.data' id
meltano select tap-smoke-test 'animals\.data' verified
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export TAP_SMOKE_TEST__SELECT='["animals\\.data.id","animals\\.data.verified"]'
```

  </TabItem>
</Tabs>


#### <a name="select-filter-extra"></a>`select_filter` extra

- Setting: `_select_filter`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__SELECT_FILTER`, e.g. `TAP_GITLAB__SELECT_FILTER`
- Equivalent to using [`meltano el`](/reference/command-line-interface#el) CLI options: `--select` and `--exclude`
- Default: `[]`

An extractor's `select_filter` [extra](/guide/configuration#plugin-extras) holds an array of stream-level filter rules
that are applied to the extractor's [discovered](/guide/integration#extractor-catalog-generation) or [provided](#catalog-extra) catalog file
when the extractor is run using [`meltano run`](/reference/command-line-interface#run), [`meltano invoke`](/reference/command-line-interface#invoke), or [`meltano el`](/reference/command-line-interface#el). These are always applied
_after_ the [schema](#schema-extra), [selection](#select-extra), and [metadata](#metadata-extra) rules are applied.

The `select_filter` extra operates at the **stream level** and determines which entire streams should be included or excluded from extraction. It preserves any property-level selections you've made with the [`select` extra](#select-extra).

## Key Differences: `select` vs `select_filter`

| `select` | `select_filter` |
|----------|----------------|
| **Stream & Property-level**: Controls which streams and properties within those streams are included | **Stream-level**: Filters which entire streams are included/excluded |
| **Purpose**: Define what data fields you want | **Purpose**: Filter streams defined in `select` to process in a specific run |
| **Preserves existing selections**: No, defines the selection | **Preserves existing selections**: Yes, filters on top of existing `select` configuration |

## When to Use `select_filter`

- **Parallelization**: Run an instance of Meltano for a specific stream with other different configuration options
- **Testing/debugging**: Temporarily include/exclude streams without changing your base configuration
- **Conditional processing**: Different runs need different subsets of your configured streams
- **CLI operations**: Via [`meltano el --select`](#el) or [`meltano el --exclude`](#el)

Selection filter rules use stream identifiers that correspond to Singer stream `tap_stream_id` values. Rules indicating that a stream should be excluded are prefixed with an exclamation mark (`!`). [Unix shell-style wildcards](<https://en.wikipedia.org/wiki/Glob_(programming)#Syntax>) can be used in stream identifiers to match multiple streams at once.

Stream names can be discovered using [`meltano select --list --all <plugin>`](/reference/command-line-interface#select).

While this extra can be managed using [`meltano config`](/reference/command-line-interface#config) or environment variables like any other setting,
selection filters are typically specified using [`meltano el`](/reference/command-line-interface#el)'s `--select` and `--exclude` options.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-gitlab
  # Base configuration: what properties we want from each stream
  select:
  - project_members     # All properties from project_members
  - commits.id          # Only id from commits
  - commits.author_name # Only author_name from commits
  - issues.*            # All properties from issues

  # Runtime filter: which streams to process (applied on top of select)
  select_filter:
  - commits             # Only process commits stream (preserves the id + author_name selection above)
  - issues              # Only process issues stream (preserves all properties selection above)
  # project_members stream will be filtered out entirely
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _select_filter '["<entity>", ...]'
meltano config <extractor> set _select_filter '["!<entity>", ...]'

meltano el <extractor> <loader> --select <entity>
meltano el <extractor> <loader> --exclude <entity>

# For example:
# Include only specific streams (most common use case)
meltano config tap-gitlab set _select_filter '["commits", "issues"]'

# Exclude specific streams
meltano config tap-gitlab set _select_filter '["!project_members", "!logs"]'

# More commonly used via CLI options:
meltano el tap-gitlab target-jsonl --select commits        # Process only commits stream
meltano el tap-gitlab target-jsonl --select commits issues # Process commits and issues streams
meltano el tap-gitlab target-jsonl --exclude logs          # Process all configured streams except logs
meltano el tap-gitlab target-jsonl --select commits --exclude archived_commits  # Include commits but exclude archived_commits

```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__SELECT_FILTER='["<entity>", ...]'
export <EXTRACTOR>__SELECT_FILTER='["!<entity>", ...]'

# For example:
# Include specific streams only
export TAP_GITLAB__SELECT_FILTER='["commits","issues"]'
# Exclude specific streams
export TAP_GITLAB__SELECT_FILTER='["!project_members","!logs"]'
```

  </TabItem>
</Tabs>

## Practical Example: Two-Phase Selection

Here's how `select` and `select_filter` work together:

```yaml
# 1. Configuration phase (select): Define what properties you want
extractors:
- name: tap-gitlab
  select:
  - users.id
  - users.name
  - commits.*
  - issues.id
  - issues.title
  - projects.*
```

```bash
# 2. Runtime phase (select_filter): Choose which streams to process
# This command will extract:
# - users: only id and name properties (from select configuration)
# - commits: all properties (from select configuration)
# - issues stream is filtered out entirely
meltano el tap-gitlab target-postgres --select users commits
```

#### `state` extra

- Setting: `_state`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__STATE`, e.g. `TAP_GITLAB__STATE`
- [`meltano el`](/reference/command-line-interface#el) CLI option: `--state`
- Default: None

An extractor's `state` [extra](/guide/configuration#plugin-extras) holds a path to a [state file](https://hub.meltano.com/singer/spec#state-files) (relative to the [project directory](/concepts/project)) to be provided to the extractor
when it is run as part of a pipeline using [`meltano el`](/reference/command-line-interface#el).

If a state path is not set, the state will be [looked up automatically](/guide/integration#incremental-replication-state) based on the ELT run's State ID.

While this extra can be managed using [`meltano config`](/reference/command-line-interface#config) or environment variables like any other setting,
a state file is typically provided using [`meltano el`](/reference/command-line-interface#el)'s `--state` option.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-gitlab
  state: extract/tap-gitlab.state.json
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _state <path>

meltano el <extractor> <loader> --state <path>

# For example:
meltano config tap-gitlab set _state extract/tap-gitlab.state.json

meltano el tap-gitlab target-jsonl --state extract/tap-gitlab.state.json
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__STATE=<path>

# For example:
export TAP_GITLAB__STATE=extract/tap-gitlab.state.json
```

  </TabItem>
</Tabs>

#### <a name="cache-catalog-extra"></a>`use_cached_catalog` extra

- Setting: `_use_cached_catalog`
- [Environment variable](/guide/configuration#configuring-settings): `<EXTRACTOR>__USE_CACHED_CATALOG`, e.g. `TAP_GITLAB__USE_CACHED_CATALOG`
- Default: `True`

An extractor's `use_cached_catalog` [extra](/guide/configuration#plugin-extras) is a boolean flag that, when set to `False`, disables the use of a cached catalog file during the extractor's discovery process. By default, Meltano will cache the catalog file generated by an extractor to speed up subsequent runs. However, if the extractor's schema has changed in a way that would affect discovery output, you may want to bypass the cache to ensure the latest catalog is used.

Setting this extra to `False` forces the extractor to perform discovery and generate a new catalog file every time it runs, which can be useful during development or when an extractor supports dynamic catalog discovery, such as in [`tap-salesforce`](https://github.com/MeltanoLabs/tap-salesforce).

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
extractors:
- name: tap-gitlab
  use_cached_catalog: false
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <extractor> set _use_cached_catalog false

# For example:
meltano config tap-gitlab set _use_cached_catalog false
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <EXTRACTOR>__USE_CACHED_CATALOG=false

# For example:
export TAP_GITLAB__USE_CACHED_CATALOG=false
```

  </TabItem>
</Tabs>

### Loaders

Loaders are [pip packages](https://pip.pypa.io/en/stable/) used by [`meltano el`](/reference/command-line-interface#el) as part of [data integration](/guide/integration).
They are responsible for loading [extracted](#extractors) data into arbitrary data destinations: databases, SaaS APIs, or file formats.

Meltano supports [Singer targets](https://singer.io): executables that implement the [Singer specification](https://hub.meltano.com/singer/spec).

To learn which loaders are [discoverable](#discoverable-plugins) and supported out of the box, refer to the [Loaders page](https://hub.meltano.com/loaders/).

#### Extras

Loaders support the following [extras](/guide/configuration#plugin-extras):

- [`dialect`](#dialect-extra)

#### `dialect` extra

- Setting: `_dialect`
- [Environment variable](/guide/configuration#configuring-settings): `<LOADER>__DIALECT`, e.g. `TARGET_POSTGRES__DIALECT`
- Default: `$MELTANO_LOADER_NAMESPACE`, which [will expand to](/guide/configuration#expansion-in-setting-values) the loader's `namespace`. Note that this default has been overridden on [discoverable](#discoverable-plugins) loaders, e.g. `postgres` for `target-postgres` and `snowflake` for `target-snowflake`.

A loader's `dialect` [extra](/guide/configuration#plugin-extras)
holds the name of the dialect of the target database, so that
[transformers](#transformers) in the same pipeline can determine the type of database to connect to.

The value of this extra [can be referenced](/guide/configuration#expansion-in-setting-values) from a transformer's configuration using the `MELTANO_LOAD__DIALECT`
[pipeline environment variable](/guide/integration#pipeline-environment-variables).
It is used as the default value for `dbt`'s `target` setting, and should therefore correspond to a target name in `transform/profile/profiles.yml`.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
loaders:
- name: target-example-db
  dialect: example-db
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <loader> set _dialect <dialect>

# For example:
meltano config target-example-db set _dialect example-db
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <LOADER>__DIALECT=<dialect>

# For example:
export TARGET_EXAMPLE_DB__DIALECT=example-db
```

  </TabItem>
</Tabs>

### Transforms

:::caution

  <p> Transform plugins are being deprecated in favor of calling dbt packages directly.</p>
  <p> The transform plugin type is still supported for now but will eventually be phased out.</p>
:::

Transforms are [dbt packages](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) containing [dbt models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models),
that are used by [`meltano el`](/reference/command-line-interface#el) as part of [data transformation](/guide/transformation).

Together with the [dbt](https://www.getdbt.com) [transformer](#transformers), they are responsible for transforming data that has been loaded into a database (data warehouse) into a different format, usually one more appropriate for [analysis](/guide/analysis).

When a transform is added to your project using [`meltano add`](/reference/command-line-interface#add),
the [dbt package Git repository](https://docs.getdbt.com/docs/building-a-dbt-project/package-management#git-packages) referenced by its `pip_url`
will be added to your project's `transform/packages.yml` and the package will be enabled in `transform/dbt_project.yml`.

#### Extras

Transforms support the following [extras](/guide/configuration#plugin-extras):

- [`package_name`](#package-name-extra)
- [`vars`](#vars-extra)

#### <a name="package-name-extra"></a>`package_name` extra

- Setting: `_package_name`
- [Environment variable](/guide/configuration#configuring-settings): `<TRANSFORM>__PACKAGE_NAME`, e.g. `TAP_GITLAB__PACKAGE_NAME`
- Default: `$MELTANO_TRANSFORM_NAMESPACE`, which [will expand to](/guide/configuration#expansion-in-setting-values) the transform's `namespace`, e.g. `tap_gitlab` for `tap-gitlab`

A transform's `package_name` [extra](/guide/configuration#plugin-extras)
holds the name of the dbt package's internal dbt project: the value of `name` in `dbt_project.yml`.

When a transform is added to your project using [`meltano add`](/reference/command-line-interface#add), this name will be added to the `models` dictionary in `transform/dbt_project.yml`.

The value of this extra [can be referenced](/guide/configuration#expansion-in-setting-values) from a transformer's configuration using the `MELTANO_TRANSFORM__PACKAGE_NAME`
[pipeline environment variable](/guide/integration#pipeline-environment-variables).
It is included in the default value for `dbt`'s `models` setting: `$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_model`.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
transforms:
- name: dbt-facebook-ads
  namespace: tap_facebook
  package_name: facebook_ads
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <transform> set _package_name <name>

# For example:
meltano config dbt-facebook-ads set _package_name facebook_ads
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <TRANSFORM>__PACKAGE_NAME=<name>

# For example:
export DBT_FACEBOOK_ADS__PACKAGE_NAME=facebook_ads
```

  </TabItem>
</Tabs>

#### `vars` extra

- Setting: `_vars`
- [Environment variable](/guide/configuration#configuring-settings): `<TRANSFORM>__VARS`, e.g. `TAP_GITLAB__VARS`
- Default: `{}` (an empty object)

A transform's `vars` [extra](/guide/configuration#plugin-extras) holds an object representing [dbt model variables](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables)
that can be referenced from a model using the [`var` function](https://docs.getdbt.com/reference/dbt-jinja-functions/var).

When a transform is added to your project using [`meltano add`](/reference/command-line-interface#add), this object will be used as the dbt model's `vars` object in `transform/dbt_project.yml`.

Because these variables are handled by dbt rather than Meltano, [environment variables](/guide/configuration#expansion-in-setting-values) can be referenced using the [`env_var` function](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var) instead of `$VAR` or `${VAR}`.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
{% raw %}
transforms:
- name: tap-gitlab
  vars:
    schema: '{{ env_var(''DBT_SOURCE_SCHEMA'') }}'
{% endraw %}
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
{% raw %}
meltano config <transform> set _vars <key> <value>

# For example
meltano config --plugin-type=transform tap-gitlab set _vars schema "{{ env_var('DBT_SOURCE_SCHEMA') }}"
{% endraw %}

```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <TRANSFORM>__VARS='{"<key>": "<value>"}'

# For example:
export TAP_GITLAB__VARS='{"schema": "{{ env_var(''DBT_SOURCE_SCHEMA'') }}"}'
```

  </TabItem>
</Tabs>

### Orchestrators

:::caution

  <p> Orchestrator plugins are transitioning over to being called Utilities. The new approach is to group all non-EL plugins under the `utility` plugin type. </p>
  <p> The orchestrator plugin type is still supported for now but will eventually be phased out as utilities take over.</p>
:::

Orchestrators are [pip packages](https://pip.pypa.io/en/stable/) responsible for [orchestrating](/guide/orchestration) a project's [scheduled pipelines](/reference/command-line-interface#schedule).

Meltano supports [Apache Airflow](https://airflow.apache.org/) out of the box, but can be used with any tool capable of reading the output of [`meltano schedule list --format=json`](/reference/command-line-interface#schedule) and executing each pipeline's [`meltano run`](/reference/command-line-interface#run) command on a schedule.

When the `airflow` utility is added to your project using [`meltano add`](/reference/command-line-interface#add),
its related [file bundle](#file-bundles) will automatically be added as well.

### Transformers

:::caution

  <p> Transformers plugins are transitioning over to being called Utilities. The new approach is to group all non-EL plugins under the `utility` plugin type. </p>
  <p> The transformer plugin type is still supported for now but will eventually be phased out as utilities take over.</p>
:::

Transformers are [pip packages](https://pip.pypa.io/en/stable/) used by [`meltano run`](/reference/command-line-interface#run) as part of [data transformation](/guide/transformation).
They are responsible for running [transforms](#transforms).

Meltano supports [dbt](https://www.getdbt.com) and its [dbt models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models) out of the box.

When the `dbt` transformer is added to your project using [`meltano add`](/reference/command-line-interface#add),
its related [file bundle](#file-bundles) will automatically be added as well.

### File bundles

File bundles are [pip packages](https://pip.pypa.io/en/stable/) bundling files you may want in your project.

When a file bundle is added to your project using [`meltano add`](/reference/command-line-interface#add),
the bundled files will automatically be added as well.
The file bundle itself will not be added to your [`meltano.yml` project file](/concepts/project#meltanoyml-project-file) unless it contains files that are
[managed by the file bundle](#update-extra) and to be updated automatically when [`meltano upgrade`](/reference/command-line-interface#upgrade) is run.

#### `update` extra

- Setting: `_update`
- [Environment variable](/guide/configuration#configuring-settings): `<BUNDLE>__UPDATE`, e.g. `DBT__UPDATE`
- Default: `{}` (an empty object)

A file bundle's `update` [extra](/guide/configuration#plugin-extras) holds an object mapping file paths (of files inside the bundle, relative to the project root) to booleans. [Glob](<https://en.wikipedia.org/wiki/Glob_(programming)>) patterns are supported to allow matching of multiple files with a single path.

When a file path's value is `True`, the matching files are considered to be managed by the file bundle and updated automatically when [`meltano upgrade`](/reference/command-line-interface#upgrade) is run.

##### How to use

Manage this extra:

<Tabs className="meltano-tabs" queryString="meltano-tabs">
  <TabItem className="meltano-tab-content" value="meltano.yml" label="meltano.yml" default>

```yaml
files:
- name: dbt
  update:
    transform/dbt_project.yml: false
    profiles/*.yml: true
```

:::info

  <p>If a file path starts with a <code>%2A</code>, it must be wrapped in quotes to be considered valid YAML. For example, using <code>%2A.yml</code> to match all <code>.yml</code> files:</p>
<pre>
files:
- name: dbt
  update:
    '*.yml': true
</pre>
:::

  </TabItem>
  <TabItem className="meltano-tab-content" value="terminal" label="terminal">

```bash
meltano config <bundle> set _update <path> <true/false>

# For example:
meltano config --plugin-type=files dbt set _update transform/dbt_project.yml false
meltano config --plugin-type=files dbt set _update profiles/*.yml true
```

  </TabItem>
  <TabItem className="meltano-tab-content" value="env" label="env">

```bash
export <BUNDLE>__UPDATE='{"<path>": <true/false>}'

# For example:
export DBT__UPDATE='{"transform/dbt_project.yml": false, "profiles/*.yml": true}'
```

  </TabItem>
</Tabs>

### Utilities

The utility plugin type represents all non-EL plugins.
Plugins that were under the transformer (e.g. dbt) and orchestrator (e.g. Airflow, Dagster) plugin types are now included as utilities.

Also any additional [pip package](https://pip.pypa.io/en/stable/) that exposes an executable can be added to your project as a utility.
Meltano has a selection of available utilities listed on [MeltanoHub](https://hub.meltano.com/utilities), or you can easily add your own custom utility.

Meltano also has an [Extension Developer Kit (EDK)](/guide/advanced-topics#extension-developer-kit-edk) that can be used to integrate existing data tools with Meltano.

#### Custom Utilities

Any [pip package](https://pip.pypa.io/en/stable/) that exposes an executable can be added to your project as a custom utility.

```bash
meltano add --custom utility <plugin>

# For example:
meltano add --custom utility yoyo
(namespace): yoyo
(pip_url): yoyo-migrations
(executable): yoyo
```

You can then invoke the executable using [`meltano invoke`](/reference/command-line-interface#invoke):

```bash
meltano invoke <plugin> [<executable arguments>...]

# For example:
meltano invoke yoyo new ./migrations -m "Add column to foo"
```

The benefit of doing this as opposed to adding the package to `requirements.txt` or running `pip install <package>` directly is that any packages installed this way benefit from Meltano's [virtual environment](https://docs.python.org/3/glossary.html#term-virtual-environment) isolation.
This avoids dependency conflicts between packages.

### Mappers

Mappers allow you to transform or manipulate data after extraction and before loading. Common applications include:

- Streams/properties can be aliased to provide custom naming downstream.
- Stream records can be filtered based on any user-defined logic.
- Properties can be transformed inline (i.e. converting types, sanitizing PII data).
- Properties can be removed from the stream.
- New properties can be added to the stream.

Note that mappers are currently only available when using [`meltano run`](/reference/command-line-interface#run).

##### How to use

You can install mappers like any other other plugin using [`meltano add`](/reference/command-line-interface#add):

```bash
$ meltano add mapper transform-field
2024-01-01T00:25:40.604941Z [info     ] Installing mapper 'transform-field'
2024-01-01T00:25:53.152127Z [info     ] Installed mapper 'transform-field'

To learn more about mapper 'transform-field', visit https://github.com/transferwise/pipelinewise-transform-field
```

Mappers are unique in that after install you don't invoke them directly. Instead you define `mappings` by name and add a config object for each mapping.
This config object is passed to the mapper when the **mapping name** is called as part of a [`meltano run`](/reference/command-line-interface#run) invocation.
Note that this differs from other plugins, as you're not invoking a plugin name - but referencing the mapping name instead.
Additionally, the requirements for the config object itself will vary by plugin.

So given a mapper with mappings configured like so:

```yaml
mappers:
  - name: transform-field
    variant: transferwise
    pip_url: pipelinewise-transform-field
    executable: transform-field
    mappings:
    - name: hide-gitlab-secrets
      config:
        transformations:
          - field_id: "author_email"
            tap_stream_name: "commits"
            type: "MASK-HIDDEN"
          - field_id: "committer_email"
            tap_stream_name: "commits"
            type: "MASK-HIDDEN"
    - name: null-created-at
      config:
        transformations:
          - field_id: "created_at"
            tap_stream_name: "accounts"
            type: "SET-NULL"
```

You can then invoke the mappings by name:

```bash

# hide-gitlab-secrets will resolve to mapping with the same name. In this case, that mapping will perform two actions.
# Transform the "author_email" field in the "commits" stream and hide the email address.
# Transform the "committer_email" field in the "commits" stream and hide the email address.
$ meltano run tap-gitlab hide-gitlab-secrets target-jsonl

# null-created-at will resolve to mapping with the same name. In this case, that mapping will perform one action.
# Transform the "created_at" field in the "accounts" stream and set it to null.
$ meltano run tap-someapi null-created-at target-jsonl
```

You can also invoke multiple mappings at once in series:

```bash
$ tap-someapi fix-null-id fix-country-code target-jsonl
```

Each mapping will execute in a unique process instance of the mapper plugin. That means that you can also
call mappings that leverage the same plugin at multiple locations numerous times within the run invocation:

```bash

# Fix any null country codes using transform-field mapper.
# Set the customers region based on their country code using your own mapper.
# Mask the id if the customer is in the EU region using transform-field mapper.
$ tap-someapi fix-null-country set-region-from-country  mask-id-if-eu target-jsonl
```

<script src="/js/tabs.js"></script>
